Multivariate Models
IS 5150/6110
Project Update
- Add me and Holden as collaborators on each of your project repositories
- Create the PR for that week’s branch to submit the associated milestone
- Once possible changes have been made to the milestone, we will merge
- Access to data should be a serious focus at this point in the semester
- However, you could use experimental or observational data
Probabilistic Perspective
- Using probability to quantify uncertainty about what we know
- All unknowns are random variables (i.e., can have probability distributions)
- We have joint distributions \(p(x, y)\) and marginal distributions \(p(x)\)
- Lots of moments or summary statistics, but they have limited information
- Bayes’ rule provides a principled way to update what we know
\[ \LARGE{p(\theta | X) \propto p(X | \theta) \ p(\theta)} \]
\[ \LARGE{\text{posterior} \propto \text{likelihood} \times \text{prior}} \]
Frequentist
\[ \theta = \text{some value} \\ X \sim \text{some distribution} \]
Bayesian
\[ \theta \sim \text{some distribution} \\ X = \text{some value} \]
Frequentist
\[ \theta = \text{some } \color{red}{\text{unknown}} \text{ value} \\ X \sim \text{some distribution} \]
Bayesian
\[ \theta \sim \text{some distribution} \\ X = \text{some } \color{red}{\text{known}} \text{ value} \]
Multivariate Models
What questions do you have?
- \(x\) scalars
- \(\mathbf{x}\) vectors (i.e., column vectors)
- \(\mathbf{X}\) matrices, composed of rows and columns
- Bonus: Python apparently fills matrices by rows
Covariance is a multivariate generalization of variance. Correlation is a normalized covariance (i.e., scaled to \([-1, 1]\)).
\[ \text{Cov}[\mathbf{x}] = \mathbf{\Sigma} = \begin{bmatrix} \text{Var}[X_1] & \text{Cov}[X_1, X_2] & \text{Cov}[X_1, X_3] \\ \text{Cov}[X_2, X_1] & \text{Var}[X_2] & \text{Cov}[X_2, X_3] \\ \text{Cov}[X_3, X_1] & \text{Cov}[X_3, X_2] & \text{Var}[X_3] \end{bmatrix} \]
Multivariate models are generalization of their univariate counterparts. Why do we need the added complexity? What is a generalized linear model?
- Multivariate Gaussian/Normal (including Bivariate)
- Exponential family (maximum entropy models)
- Mixture models (oh my!)
We can express any probabilistic model (i.e., joint distribution) as a graph where conditional independence and causal structure is encoded.
- Empty nodes are observed
- Shaded nodes are unobserved
- Edges are the lines connecting nodes
- Arrows show the flow of information (causality)
- We are especially interested in directed acyclic graphs
::::
